Selection of Relevant Features in Machine LearningPat

In this paper, we review the problem of selecting relevant features for use in machine learning. We describe this problem in terms of heuristic search through a space of feature sets, and we identify four dimensions along which approaches to the problem can vary. We consider recent work on feature selection in terms of this framework, then close with some challenges for future work in the area. The selection of relevant features, and the elimination of irrelevant ones, is a central problem in machine learning. Before an induction algorithm can move beyond the training data to make predictions about novel test cases, it must decide which attributes to use in these predictions and which to ignore. Intuitively, one would like the learner to use only those attributes that arèrelevant' to the target concept. There have been a few attempts to deenèrelevance' in the context of machine learning, as John, Kohavi, and PPeger (1994) have noted in their review of this topic. Because we will review a variety of approaches, we do not take a position on this issue here. We will focus instead on the task of selecting relevant features (however deened) for use in learning and prediction. Many induction methods attempt to deal directly with the problem of attribute selection, especially ones that operate on logical representations. For instance, techniques for inducing logical conjunctions do little more than add or remove features from the concept description. Addition and deletion of single attributes also constitute the basic operations of more sophisticated methods for inducing decision lists and decision trees. Some nonlogical induction methods, like those for neural networks and Bayesian classiiers, instead use weights to assign degrees of relevance to attributes. And some learning schemes, such as the simple nearest neighbor method, ignore the issue of relevance entirely. We would like induction algorithms that scale well to domains with many irrelevant features. More specifically , we would like the sample complexity (the number of training cases needed to reach a given level of accuracy) to grow slowly with the number of irrelevant attributes. Theoretical results for algorithms that search restricted hypothesis spaces are encouraging. For instance, the worst-case number of errors made by Littlestone's (1987) Winnow method grows only logarithmically with the number of irrelevant features. Pazzani and Sarrett's (1992) average-case analysis for Wholist, a simple conjunctive algorithm, and Lang-ley and Iba's (1993) treatment of the naive Bayesian classiier, suggest …